REPORT  DOCUMENTATION  PAGE 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  inclu> 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information  St 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  bervi 
Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Buaget,  Pap 


AGENCY  USE  ONLY  ( Leave  blank) 


REPORT  DATE 

3  SEP  03 


afrl-sr-ar-tr-03- 


3.  REPC  . . 

FINAL  REPORT  -  15  MAY  97  TO  14  MAY  02 


4.  TITLE  AND  SUBTITLE 

5.  FUNDING  NUMBERS 

(MURI  97)  INTEGRATED  COGNITIVE,  COMPUTATIONAL  AND  BIOLOGICAL 
ASSESSMENT  OF  WORKLOAD  IN  DECISION  MAKING 

F49620-97- 1-0368 

6.  AUTHOR(S) 

3484/US 

PROF.  MARCEL  A.  JUST 

61103D 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

CARNEGIE  MELLON  UNIVERSITY 

DEPARTMENT  OF  PSYCHOLOGY 

PITTSBURGH,  PA  15213 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

AFOSR/NL 

4015  WILSON  BLVD.,  SUITE  713 

ARLINGTON,  VA  22203-1954 

11.  SUPPLEMENTARY  NOTES 

12a.  DISTRIBUTION  AVAILABILITY  STATEMENT 


12b.  DISTRIBUTION  CODE 


APPROVE  FOR  PUBLIC  RELEASE:  DISTRIBUTION  UNLIMITED. 


13.  ABSTRACT  ( Maximum  200  words) 

The  project  research  is  completed.  Several  manuscripts  are  still  in  the  process  of  being  prepared  for  publicaiton.  The  project 
has  developed  paradigms  to  study  high-level,  high-demand  cognitive  tasks  with  fMRI.  The  project  has  discovered  a  surprising 
phenomenon  regarding  a  mutual  constraint  on  the  total  amount  of  cortical  activation  possible  in  a  dual  task  situation.  The 
project  has  determined  the  effects  of  input  modality  in  dual  tasking.  The  project  has  developed  an  experimental  control 
system  (CogLab  NT)  that  will  enable  future  fMRI  investigation  of  the  performance  of  dynamic  tasks  with  some  of  the  critical 
properties  of  high-information-load  STE’s.  The  project  has  developed  paradigms  to  study  high-level,  high-demand  dynamic 
cognitive  tasks  behaviorally.  The  project  has  developed  brain-based  measures  of  cognitive  workload.  The  project  has 
dveloped  a  theory  of  learning  in  dynamic  decision  environments,  called  Instance-Based  Learning  Theory  (IBLT). 


14.  SUBJECT  TERMS 


20031006  061 


115.  NUMBER  OF  PAGES 


116.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION  |  18.  SECURITY  CLASSIFICATION  19.  SECURITY  CLASSIFICATION  20.  LIMITATION  OF  ABSTRACT 
OF  REPORT  OF  THIS  PAGE  OF  ABSTRACT 


UNCLAS 


UNCLAS 


UNCLAS 


Standard  Form  298  (Rev,  2-89)  (EG) 

Prescribed  by  ANSI  Std.  239.18  ^  _ 

Designed  using  Perform  Pro,  WHS/DIOR,  Oct  94 


^  J  ti  s  ? 


c:\documents  and  settings\sorkin\local  settings\temporary  internet  files\olkdl\muri_final-report.doc  09/03/03 


Air  Force  Office  of  Scientific  Research 
Final  Report 
August  2003 


f 


TITLE: 


GRANT 

NUMBER 

F49620-97- 1  -0368 

Duration  of  study: 

5/15/97-  4/30/02 

_ 

PRINCIPAL  INVESTIGATOR(S): 

Marcel  Adam  Just,  Ph.D.  . 

Co-INVESTIGATOR(S): 

Patricia  A.  Carpenter,  Ph.D. 

Cleotilde  Gonzalez,  Ph.D. 

Javier  Lerch.  Ph.D. 

INSTITUTION: 

Carnegie  Mellon  University 

MAILING  ADDRESS: 

Department  of  Psychology 

Carnegie  Mellon  University 

Pittsburgh,  PA  15213 

EMAIL 

Just(a>,cmu.edu 

PHONE: 

(412)  268-2791 

PAX: 

(412)  268-2804 

Excerpt  from  Original  Project  Abstract,  describing  only  the  funded  parts  of  the  project 

The  proposed  projects  focus  on  the  measurement  of  cognitive  workload  and  the  task  conditions  that 
lead  to  it  in  synthetic  task  environments  (STE’s)  adapted  from  DoD  labs,  in  combination  with  other 
synthetic  tasks  that  have  been  psychometrically  validated  with  performance  in  real-world  decision-making 
environments  (e.g.  911  operators,  air  traffic  control  tasks,  real-time  resource  allocation  in  complex  mail 
sorting  environment).  The  research  concentration  areas  of  the  program  include  cognitive  workload 
measurement,  cognitive  modeling  and  task  analysis.  The  disciplinary  approaches  that  will  be  integrated 
include:  psychological  science,  computational  modeling,  neuroscience  (functional  brain  imaging). 

The  proposed  program  includes  four  research  projects  that  address  the  nature  and  impact  of  cognitive 
workload  on  complex,  real-time  decision-making,  each  addressing  several  of  the  research  concentration 
areas. 

Project  1:  Neuroimaging  of  Cognitive  Workload  during  Decision  Making 
Project  3:  Adaptation  in  Real-Time  Dynamic  Decision  Making 

The  research  approaches  include:  (a)  the  use  of  STE’s  developed  by  DoD  labs  or  tasks  that  have  been 
validated  against  real-world  complex  decision  making,  (b)  the  development  of  task  analyses  to  characterize 
the  STE’s,  (c)  collecting  detailed  process  tracing  data  (information  search,  eye  tracking,  keystroke  logs, 
timing  of  decisions,  verbal  protocols  to  learn  about  decision  heuristics),  (d)  the  use  of  personnel 
classification  tests  to  investigate  individual  differences  in  decision  making  under  workload,  (e)  the  use  of 
biological  and  neurological  (brain  imaging)  measures  of  workload  effects,  and  (f)  cognitive  modeling  of 
information  search  strategies  and  decision  heuristics. 
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2.  Objectives: 


Project  1:  Neuroimaging  of  Cognitive  Workload  during  Decision  Making 

1.  To  discover  attributes  of  brain  function  that  are  indices  of  cognitive  workload 

2.  To  develop  quantitative,  automated  measures  of  relevant  brain  activation  attributes 

3.  To  develop  a  theory  and  computational  model  that  links  the  cognitive  and  cortical  levels 

[There  is  no  Project  2] 

Project  3:  Adaptation  in  Real-Time  Dynamic  Decision  Making 

1.  To  develop  a  theory  of  adaptation  for  individuals  working  in  real-time  dynamic  decision  making  (DDM) 
tasks.  The  theory  should  be  able  to  predict  and  explain: 

•  the  effects  of  workload  in  DDM  in  the  short  and  long  term. 

•  the  learning  process  in  DDM  tasks 

3.  Status  of  Effort: 

The  project  research  is  completed.  Several  manuscripts  are  still  in  the  process  of  being  prepared  for 
publication. 

4.  Accomplishments/New  Findings: 

•  The  project  has  developed  paradigms  to  study  high-level,  high-demand  cognitive  tasks  with  fMRI. 

•  The  project  has  discovered  a  surprising  phenomenon  regarding  a  mutual  constraint  on  the  total  amount 
of  cortical  activation  possible  in  a  dual  task  situation. 

•  The  project  has  determined  the  effects  of  input  modality  in  dual  tasking. 

•  The  project  has  developed  an  experimental  control  system  (CogLab  NT)  that  will  enable  future  fMRI 
investigation  of  the  performance  of  dynamic  tasks  with  some  of  the  critical  properties  of  high- 
information-load  STE’s. 

•  The  project  has  developed  paradigms  to  study  high-level,  high-demand  dynamic  cognitive  tasks 
behaviorally. 

•  The  project  has  developed  brain-based  measures  of  cognitive  workload. 

•  The  project  has  developed  a  theory  of  learning  in  dynamic  decision  environments,  called  Instance- 
Based  Learning  Theory  (IBLT). 


Project  1  narrative:  Neuroimaging  of  Cognitive  Workload  during  Decision  Making 

Project  1  developed  processing  models  and  analytic  methods  in  the  context  of  brain  imaging  studies  of  dual 
task  performance.  These  were  particular  kinds  of  dual  tasks,  namely  two  high-level  tasks  that  were  being 
performed  concurrently,  rather  than  by  rapidly  switching  attention  between  two  tasks.  The  prototype 
paradigm  required  participants  to  listen  to  sentences  and  judge  them  as  true  or  false  while  simultaneously 
performing  a  mental  rotation  task,  to  judge  whether  one  complex  3-dimension  object  could  be  rotated  into 
another.  The  two  tasks  used  different  input  modalities  and  a  different  response  apparatus.  The  paradigm 
was  adapted  from  the  Multi  Attribute  Test  (MAT).  The  intent  was  to  focus  the  resource  competition  on  the 
high-level  cognition. 
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Decreased  brain  activation  in  concurrent  task  performance.  The  most  important  empirical  outcome  was 
that  there  was  much  less  brain  activation  associated  with  a  task  when  it  was  one  of  two  dual  tasks  than 
when  it  was  being  performed  alone.  This  was  a  major  scientific  contribution  and  with  considerable 
practical  implications.  The  scientific  article  was  published  in  Neuroimage,  and  was  also  featured  in  an 
article  in  the  New  York  Times  and  many  other  newspapers  and  periodicals.  The  most  widely-discussed 
implication  of  the  study  is  that  cell  phone  use  (the  engagement  in  conversation,  not  the  dialing)  during 
driving  might  decrease  the  activation  associated  with  driving.  More  generally,  the  results  suggest  that  the 
bottleneck  in  high  level  concurrent  task  performance  is  a  limitation  on  brain  resources  for  cognition. 


The  mean  number  of  voxels  activated  in  each  of  the  two  single-task  conditions  and  in  the  dual 
condition,  in  the  language  areas  (left  and  right  superior  temporal  gyrus,  cross-hatched  bars)  and  in 
the  spatial  processing  areas  (left  and  right  parietal  lobule,  open  bars)  (and  standard  errors  of  the 
means  over  18  participants). 

Other  bottlenecks  can  occur  in  addition,  both  in  input  and  output,  but  those  bottlenecks  can  in  principle  be 
circumvented.  The  bottleneck  on  central  resources  for  cognition  cannot  be  avoided  by  re-designing  the  task 
environment.  The  only  hope  for  overcoming  this  central  bottleneck  is  to  make  the  task  performance  in  the 
two  component  tasks  less  demanding  of  resources. 

Temporal  relations  between  two  concurrent  tasks.  Two  studies  examined  the  importance  of  the  precise 
temporal  relations  between  the  cognitive  demands  in  the  two  concurrent  tasks.  One  study  made  the  peak 
demands  in  the  two  tasks  occur  at  the  same  time  (placing  the  two  tasks  in  phase  with  each  other)  versus  out 
of  phase  with  each  other.  Having  the  two  tasks  out  of  phase  was  more  beneficial,  in  that  there  was  more 


brain  activation  for  each  of  the  component  tasks  in  the  out-of-phase  condition.  This  finding  shows  that  the 
limitation  on  central  cognitive  resources  is  time  sensitive,  and  a  time-shift  of  even  a  few  seconds  can 
ameliorate  the  concurrent  processing  situation.  (Of  course,  in  some  dynamic  tasks,  such  as  high-speed 
driving,  such  time-shifting  is  generally  not  possible.  In  that  case,  it  is  all  the  more  important  to  time-shift  a 
secondary  task  whose  execution  time  is  less  critical). 

A  second  study  in  this  area  examined  the  temporal  relations  in  more  detail,  to  determine  the  effect  of  the 
onset  of  a  second  task  during  the  performance  of  a  first  task.  The  two  tasks  involved  were  listening  to  a 
single  sentence  (a  more  automatic  task)  and  performing  a  single  mental  rotation  task  item  (a  more 
controlled  and  less  automatic  task).  The  onset  of  the  two  stimuli  differed  by  2  sec.  The  finding  was  that  it 
requires  more  brain  work  to  interrupt  the  more  automatic  task.  The  frontal  executive  system  (DLPFC) 
showed  greater  activation  for  the  Sentence  first  condition  than  for  the  Rotation  first  condition.  The 
manuscript  describing  this  study  is  being  prepared  for  publication. 

Stimulus  saliency  in  dual  visual  tasks.  One  of  the  studies  examined  how  the  relative  visual  saliency  of 
two  superimposed  stimuli  determined  the  brain  response  to  each  of  the  components.  The  processing  of  two 
superimposed  visual  stimuli  is  a  little  like  viewing  a  head-up  display  superimposed  on  a  windshield  view. 
The  experiment  manipulated  the  relative  contrast  (and  hence  saliency)  of  the  two  superimposed  stimuli. 

One  of  the  stimuli  was  a  picture  of  a  house  and  the  other  was  a  picture  of  a  face.  The  reason  for 
superimposing  these  particular  types  of  stimuli  is  that  the  brain  response  to  each  one  alone  is  fairly  well 
known  Our  new  study  showed  that  the  amount  of  activation  in  each  brain  area  varies  with  the  relative 
amount  of  contrast  (visual  saliency)  of  the  each  of  the  stimuli.  For  example,  there  is  an  area  in  the  fusiform 
cortex  that  typically  responds  strongly  to  a  picture  of  a  face.  If  a  person  is  viewing  a  face-house 
superimposition,  then  the  activation  response  in  the  fusiform  area  is  proportional  to  the  visual  saliency  of 
the  face  in  the  superimposition.  Thus  the  brain  response  in  this  dual  stimulus  situation  is  governed  in  part 
by  their  relative  saliency  in  the  display.  An  engineered  system  could  determine  the  brain  response  to  one 
component  of  a  superimposed  display  by  varying  its  relative  contrast.  The  manuscript  describing  this  study 
is  being  prepared  for  publication. 

Strategic  control  over  brain  resource  allocation.  One  study  asked  the  participants  to  differentially 
allocate  their  attention  to  two  concurrent  tasks,  under  their  own  strategic  control.  The  two  tasks  involved 
were  listening  to  a  single  sentence  (a  more  automatic  task)  and  performing  a  single  mental  rotation  t&sk 
item  (a  more  controlled  and  less  automatic  task).  The  most  general  result  was  that  there  was  some  ability  to 
allocate  control  brain  resources  under  strategic  control,  but  the  size  of  the  effect  was  modest,  having  about 
a  25%  impact  on  the  amount  of  brain  activation  difference  between  trying  to  give  a  task  major  attention 
versus  secondary  attention.  A  more  specific  result  was  that  there  was  a  greater  ability  to  control  the  brain 
resource  allocation  for  the  less  automatic  task  (rotation)  than  for  the  automatic  task  (sentence  listening). 
That  is,  it  is  difficult  to  get  one’s  brain  to  stop  processing  an  incoming  sentence.  Trying  to  “gate  out”  a 
spoken  sentence  has  relatively  little  effect  on  the  brain  activation  associated  with  the  sentence  processing 
(particularly  in  the  left  posterior  temporal  area  (roughly  Wernicke  s  area)  and  in  the  primary  auditory . 
cortex  (Heschl’s  gyrus)).  By  contrast,  one  can  ignore  an  effortful  task  like  mental  rotation.  These  findings 
indicate  that  due  consideration  should  be  given  to  what  sights  and  sounds  a  computer  interface  presents  in  a 
decision-making  situation.  Presenting  a  stimulus  that  is  processed  automatically  initiates  a  near-mandatory 
processing. 

Effects  of  training  in  dual  tasks.  One  of  the  brain  imaging  studies  contrasted  two  forms  of  training  that 
should  make  the  processing  more  automatic.  The  training  was  either  on  the  two  tasks  performed  alone 
(individual  task  training)  or  on  the  two  tasks  performed  simultaneously  (dual  task  training).  The  outcome 
was  similar  for  the  two  types  of  training. 
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Planning  in  a  problem-solving  task.  A  series  of  brain  imaging  studies  examined  the  decision-making 
processes  in  a  puzzle  task  called  the  Tower  of  London.  A  distinguishing  property  of  this  task  is  that  all  of 
the  moves  (up  to  6,  in  our  studies)  have  to  be  planned  in  advance  to  ensure  that  they  conclude  in  a  shortest- 
path  solution.  One  of  the  main  results  was  the  difference  in  cortical  activation  between  the  planning  of 
easier  versus  harder  problems.  The  simpler  problems  required  the  planning  of  fewer  moves  (2-3)  whereas 
the  harder  problems  required  the  planning  of  4-6  moves.  (The  problems  with  more  moves  typically  entail 
the  setting  of  subgoals).  For  the  simpler  problems,  the  cortical  control  produced  parietal  activation,  whereas 
the  harder  problems  produced  additional  frontal  (DLPFC)  activation.  The  significance  of  this  finding  is  that 
simpler  decisions  can  be  made  on  the  basis  of  a  perceptual  analysis,  whereas  the  more  complex  decisions 
require  the  clear  involvement  of  a  prefrontal  cortex  planning  capability.  This  finding  reflects  the  more 
general  view  of  the  cortical  system  as  dynamically  adapting  to  the  task  demands  as  they  arise.  That  is,  the 
cortical  centers  are  activated  on  an  as-needed  basis,  a  dynamic  resource-allocation  system  for  neural 
resources.  A  computational  model  for  this  task  provided  a  precise  account  for  the  behavioral  and  brain 
imaging  results.  Some  of  the  outcomes  of  the  project  have  appeared  in  Neuropsychologia.  (This  project  was 
also  supported  in  part  by  ONR). 

Summary  of  empirical  findings.  The  project  started  out  by  developing  new  STE’s  (synthetic  task 
environments)  suitable  for  investigating  dynamic  decision  making,  including  the  development  of  software, 
hardware,  and  new  data  processing  techniques,  for  both  brain  imaging  and  behavioral  studies.  This  new 
research  infrastructure  was  then  put  to  use  into  investigating  various  facets  of  dynamic  decision-making 

The  results  suggest  that  there  is  an  upper  limit  on  the  total  amount  of  cortical  activation  that  can  be 
supported  at  one  time.  This  corresponds  to  the  difficulty  in  fully  attending  to  two  streams  of  thought  at  one 
time.  But  the  phenomenon  is  better  described  as  a  resource  constraint. 

This  new  interpretation  has  several  interesting  implications.  First,  it  suggests  that  perhaps  two  streams  of 
thought  could  be  attended  if  they  one  or  both  of  them  were  undemanding.  Consistent  with  this  view, 
driving  and  conversing  can  come  to  be  co-perfbrmed  as  driving  skills  become  automated  and  less  resource 
demanding  than  they  are  in  a  novice  driver.  A  second  implication  is  that  extensive  training  may  automate 
performance  of  two  tasks  sufficiently  to  make  them  co-performable.  However,  even  for  an  experienced 
driver,  a  complex  and  rapidly  changing  traffic  situation  can  put  an  end  to  conversation,  so  training  probably 
is  a  relative  rather  than  an  absolute  solution. 

Cognitive  workload  principles  emerging  from  Project  1.  The  research  has  enabled  the  development  and 
refinement  of  the  following  principles  of  cognitive  workload 

•  performance  is  capacity-constrained 

•  attention  is  allocated  dynamically 

•  in  a  given  task,  activation  is  distributed  in  the  brain  in  10-20  discrete  centers 

•  there  are  individual  differences  in  performance  (beyond  storage  and  processing  speed),  predicted  by 
psychometric  test  scores 

•  Task  requirements  (time  course  of  load)  determine  level  of  performance  and  rate  of  learning 


Project  3  narrative:  Adaptation  in  Real-Time  Dynamic  Decision  Making 

The  three  major  products  of  this  research  were: 

•  a  set  of  Synthetic  Task  Environments  (STEs)  used  as  tools  for  research  DDM  tasks 

•  results  from  two  major  experiments  to  test  the  effects  of  workload  and  pace  of  learning  in  DDM 
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•  a  theory  of  learning  in  dynamic,  complex  situations,  developed  using  computational/cognitive 
modeling. 

Three  STEs  (Synthetic  Task  Environments)  were  developed  and  used  in  this  research.  The  first  STE  is  an 
Automated  Mail  Sorting  Task  (AMST)  developed  from  real  world  observations  in  the  US  Post  Office 
(Lerch  and  Harter,  2001).  The  second  STE  is  an  isomorph  of  AMST  that  is  more  flexible,  and  it  provides 
the  experimenter  with  multiple  manipulations  and  definitions  of  scenarios.  This  is  called  the  Water  ^ 
Purification  Plant  (WPP),  a  task  that  presents  many  advantages  over  other  STEs  or  “Microworlds’  used  in 
the  study  of  DDM.  A  summary  of  DDM  STEs  and  the  criteria  for  comparison  is  presented  in  (Gonzalez 
and  Vanyukov,  submitted).  The  third  STE  was  a  Multi-Attribute  Task  (MAT)  Battery  simulation, 
originally  developed  at  NASA  by  Comstock  and  Amegard  (1992),  but  adapted  to  work  with  WPP. 

Two  main  experiments  using  WPP  and  WPP  with  MAT  demonstrated  detrimental  effects  of  task  workload. 
Workload  was  manipulated  as  time  pressure  and  multitasking  (Gonzalez  (a)  in  Press;  and  Gonzalez  (b), 
under  review).  The  experiments  showed  that  task  workload  interacts  with  the  inherent  human  cognitive 
abilities  to  produce  different  learning  curves.  Task  workload  produces  negative  effects  on  learning  more 
pronouncedly  for  individuals  with  low  fluid  intelligence.  Time  pressure  produces  poor  performance  for 
individuals  who  practice  under  time  constraints  regardless  the  length  of  practice  they  are  given.  That  is, 
best  learning  occurs  only  when  individuals  practice  with  no  time  constraints.  Time  constraints  limit 
individuals  to  generate  alternatives,  produce  poor  exploration  of  the  environment  and  therefore,  poor 
learning  in  the  long  term.  Further,  we  analyzed  micro-behavior,  comparing  individual’s  behavior  to  pre¬ 
defined  simple  heuristics.  We  found  that  people  follow  different  strategies  under  workload.  Particularly, 
individuals  with  low  cognitive  abilities  and  under  time  pressure  follow  heuristics  increasingly  over  time, 
while  individuals  with  high  cognitive  abilities  or  any  individual  under  low  workload  reduce  their  use  of 
heuristics  over  time.  The  reduction  of  the  use  of  simple  heuristics  is  associated  with  best  performance. 

This  indicates  that  individuals  learn  to  use  the  knowledge  acquired  in  the  context  of  the  task,  and  that 
taking  advantage  of  this  knowledge  leads  them  to  best  performance. 

Using  cognitive  modeling  with  ACT-R,  we  developed  a  new  theory  of  decision  making  in  dynamic 
environments  (Gonzalez,  Lerch,  and  Lebiere,  2003).  This  new  theory  is  called  Instance  Based  Learning 
Theory  (IBLT).  Traditionally,  decision  making  has  been  studied  under  two  groups:  Judgment  and  Choice. 
Often,  decision  making  theories  do  not  apply  to  real-world  decision  making.  In  the  real  world,  decision 

making  is  dynamic.  Decisions 
made  today  change  the  situations 
considered  to  make  decisions 
tomorrow,  and  the  environment 
changes  independently  of  the 
user’s  actions.  IBLT  extends  the 
traditional  view  of  decision 
making  including  the  process  of 
recognition, 
feedback,  and 

separating  the  choice  from  its 
execution.  The  figure  to  the  left 
shows  this  process. 

IBLT  relies  of  several  learning 
principles.  One  particular 
principle  is  that,  individuals  store 


Decision 


Execution 
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instances  composed  of  three  elements:  the  decision  making  situation,  the  decision  and  the  utility  of  the 
decision.  The  IBLT  process  suggests  that  these  pieces  of  knowledge  develop  over  time  and  are  retrieved 
when  future  decisions  are  made. 

IBLT  emerged  from  the  process  of  doing  a  cognitive  model  of  decision  making  in  WPP.  Observations, 
Verbal  protocols  (Gonzalez,  in  press),  and  the  previously  presented  empirical  studies  led  us  to  a  process 
that  we  could  implement  in  a  cognitive  theory  ACT-R.  The  model’s  data  compared  to  human  data.  In 
particular,  the  learning  mechanisms  proposed  in  IBLT  were  tested  against  human  data.  Some  of  the 
findings  are  that:  decision  makers  acquire  decision  instances  with  practice  in  the  task;  instance-based 
decision-making  does  not  rely  on  one  particular  previous  instance,  but  rather  on  a  blending  or  accumulation 
of  instances  from  the  past;  decision  makers  do  not  directly  respond  to  the  feedback  in  the  task,  relying  more 
on  the  recognition  process  to  learn. 

Framing  effects  in  decision  making.  We  developed  a  new  paradigm  for  studying  a  classical  decision 
making  task  using  brain  imaging  to  measure  cognitive  workload.  The  study  examined  the  neural  basis  for 
the  ubiquitous  finding  concerning  the  framing  effect  in  the  choice  between  risky  and  certain  outcomes.  The 
framing  effect  shows  that  a  description  of  a  problem  in  positive  (gains)  versus  negative  (losses)  terms, 
systematically  affects  the  risk  attitude,  even  though  the  logic  of  the  problem  is  identical  in  the  two 
statements.  For  example,  a  positive  statement  might  describe  a  medical  policy  decision  in  terms  of  how 
many  lives  are  saved  as  a  result  of  one  choice,  whereas  a  negative  statement  would  describe  the  same 
situation  in  terms  of  how  many  lives  are  lost  as  a  result  of  the  other  choice.  People  consistently  are  risk- 
averse  for  gains  and  risk-seeking  for  losses.  The  framing  effect  has  previously  been  explained  through  an 
appeal  to  economic  theories.  But  rarely  has  the  underlying  cognitive  mechanism  been  examined.  This 
project  made  an  important  new  contribution  to  understanding  this  decision-making  process  by  showing  that 
it  is  not  the  difference  between  the  positive  and  negative  frames  that  elicits  different  brain  activity.  Rather, 
it  is  the  choice  (certain  or  risky  outcome)  that  elicits  different  brain  activity.  Risk  aversion  in  gains 
manifests  itself  as  significantly  higher  amounts  of  brain  activity  (in  frontal  and  parietal  areas)  in  risky 
compared  to  certain  responses.  Risk  seeking  in  losses  manifests  itself  as  similarly  high  brain  activity  in 
both  risky  and  certain  responses.  That  is,  when  people  evaluate  losses,  both  the  certain  and  the  risky 
outcomes  involve  similar  brain  activity  in  frontal  and  parietal  areas.  But  when  they  evaluate  gains,  the 
risky  outcomes  elicit  significantly  higher  activity  than  the  certain  outcomes.  The  work  has  been  written  up 
and  submitted  for  publication  (Gonzalez,  Dana,  Koshino,  Just,  submitted  for  publication). 

Summary  of  Empirical  Findings 

•  Individuals  under  high  time  constraints  will  learn  poorly  compared  to  individuals  under  low  time 
constraints,  regardless  the  amount  of  practice. 

•  Under  high  workload  individuals  have  more  difficulty  acquiring  and  refining  decision  instances.  Less 
and  less  diverse  instances  limit  learning. 

•  Cognitive  capacity  interacts  with  the  cognitive  workload.  Individuals  with  low  cognitive  capacity  and 
under  high  cognitive  workload  have  more  difficulties  to  learn.  The  same  individuals  under  low 
cognitive  workload  have  more  time  to  acquire  and  refine  the  task  knowledge. 

•  With  learning  individuals  gradually  switch  from  rule-based  to  instance-based  performance. 

•  Cognitive  modeling  makes  the  instance-based  description  of  the  decision-making  precise  and  dynamic 

•  Differential  brain  activity  appears  for  risky  and  certain  choices.  More  brain  activity  in  parietal  and 
frontal  areas  is  related  to  risky  choices  related  to  gains. 
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Technical  development  .  UDI 

One  large  initial  success  was  in  creating  a  paradigm  for  studying  concurrent  task  processing  in  an  M 
scanner.  The  paradigm  required  the  coordination  of  MRI-compatible  earphones  with  a  visual  display.  A 
special  MRI-compatible  response  apparatus  was  developed.  Some  specialized  equipment  was  built  and 
some  of  it  was  purchased.  Specialized  software  was  developed  for  controlling  two  high  level  tasks 
concurrently.  Finally,  the  two  tasks  were  designed  so  that  it  was  possible  to  perform  them  concurrently, 
with  resource  competition  focused  at  the  level  of  central  resources. 

One  of  the  main  software  development  accomplishments  was  an  experimental  control  system  (CogLab  NT) 
for  running  high-level  cognitive  tasks,  synchronized  with  and  MRI  scanner.  A  related  accomplishment  was 
the  construction  of  an  integrated  system  of  about  8  software  systems  to  underpin  the  cognitive  bram 
imaging  research. 


COMPUTATIONAL  INFRASTRUCTURE: 
MEASURING  WORKLOAD  IN  fMRI 


CogLab  NT  controls  the  running  of  an  experiment  under  Windows  NT  or  Windows  2000  and  it  provides 
flexible  experiment  design. 

•  The  program  is  capable  of  presenting  a  variety  of  stimuli,  including/text,  images,  and  sounds. 

•  An  experiment  can  be  of  “blocked”  or  “single-item”  design,  with  a  pre-defined  sequence  of  events  or 
self-paced,  etc. 

•  It  also  records  subject’s  responses  and  synchronizing  pulses  from  the  MR  scanner. 

To  assess  the  volume  of  cortical  activation  in  a  given  brain  center,  we  developed  a  system  for  defining 
anatomical  regions  of  interest  (ROI),  and  then  measured  the  activation  within  each  independently 
(anatomically)  defined  region,  using  a  previously  established  parcellation  scheme.  We  developed 
supporting  software  that  helped  with: 

•  Coregistration  of  structural  (anatomical)  and  functional  (activation)  data 

•  BSE  (Brain  Surface  Extraction),  stripping  away  the  image  of  the  skull,  leaving  only  the  brain 

•  Transformation  into  standard  (Talairach)  space 

The  project  developed  software  (Voxcount)  for  measuring  volumetric  indices  of  brain  activation. 

•  Number  of  activated  voxels  per  ROI  (region) 

•  %  signal  intensity  change  for  active  voxels. 

An  additional  program  measured  some  of  the  dynamics  of  the  brain  activity  (Voxcor) 

•  correlation  between  the  time  series  of  the  activation  in  two  areas  as  an  estimate  of  functional 
connectivity. 

•  Functional  connectivity  increases  with  cognitive  workload 

•  Time  courses  of  active  voxels  in  different  areas. 

These  programs  were  integrated  with  other  software  tools  that  are  widely  available.  The  value  added  of  this 
project’s  software  contribution  is  that  they  enabled  the  measurement  of  cognitive  workload,  and  not  just 
locations  of  brain  activations. 

In  summary,  the  project  helped  determine  the  how  variations  in  cognitive  workload  impacted  on  real-time 
synthetic  task  environments  entailing  decision-making.  Fine  grained  measures  of  performance,  strategy 
assessments,  task  analysis,  and  cognitive  modeling  were  developed  to  provide  a  broad-ranging  account  of 
cognitive  workload  in  these  situations.  The  workload  measures  are  both  behavioral  and  biological  (brain 
imaging-based).  Software  and  hardware  for  the  synthetic  task  environments  enable  continuing  research  on 
high-information-load  tasks  of  real-world  relevance. 
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Postdoctoral  Associate 


9 


Alice  McEleney 
Sharlene  Newman 
Erik  Reichle 
Donald  E.  Harter 
Huilong  Wu 


Postdoctoral  Associate 
Postdoctoral  Associate 
Postdoctoral  Associate 
Doctoral  student 
Doctoral  student 


6.  Publications  supported  by  the  grant: 

Just,  M.  A.,  Carpenter,  P.  A.,  Keller,  T.  A.,  Emery,  L.,  Zajac,  H.,  &  Thulbom,  K.  R.  (2001). 
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perception.  Neuropsychologia,  41, 1668-1682. 


Gonzalez,  C.,  Lerch,  F.  J.,  &  Lebiere,  C.  (2003).  Instance-Based  Learning  in  Real-Time  Dynamic  Decision 
Making.  Cognitive  Science,  27,  591-635. 

Gonzalez,  C,  Learning  to  Make  Decisions  in  Dynamic  Environments:  Effects  of  Time  Constraints  and 
Cognitive  Abilities.  Human  Factors.  In  Press. 

Gonzalez,  C.  (2003).  Novice  and  expert  verbal  protocols  in  real-time  dynamic  decision-making.  Denver, 
Co.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society. 


Submitted  Manuscripts: 

Kravitz,  D.  J.,  &  Just,  M.  A.  Distributed  cortical  activity  in  object  recognition:  Graded  activation  along  a 
house/face  continuum. 

McEleney,  A.,  Newman,  S.  D.,  Keller,  T.  A.,  &  Just,  M.  A.  Deductive  reasoning  and  the  brain:  An  fMRI 
study  of  conditional  reasoning. 

Gonzalez,  C.  &  Vanyukov,  P.  Use  of  microworlds  to  study  the  psychology  of  dynamic  decision  making. 
Gonzalez,  C.  Decision  making  in  a  dynamic,  real-time  environment:  learning  under  high  cognitive 
workload. 


7.  Interactions/Transitions: 


a.  Presentations: 


Invited  Presentations 


Measuring  brain  function  in  high-tech  environments:  fMRI  studies  of  cognitive  workload  during  real-time 
dynamic  decision  making.  Invited  presentation  at  the  Army  War  College  on  the  Impact  of  Technology  on 
the  “Physiological  and  Psychological  Dimensions  of  Warfare’  at  Georgia  Tech,  December  9-10, 1997, 
Atlanta,  Georgia. 

Cognitive  loading  and  the  future  military.  Invited  presentation  at  the  Cognitive  Sciences  Workshop 
sponsored  by  the  Science  Applications  International  Corporation  (S  AIC),  January  8-9, 1 998,  Strategic 
Assessment  Center,  McLean,  Virginia. 
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fMRI  and  the  Neuro-architecture  of  cognition.  Invited  presentation  at  the  National  Institute  of  Mental 
Health  “Cognitive  Neuroimaging  Research:  Design  and  Interpretation  Meeting”,  September  13-15,  1998, 
Rockville,  Maryland. 

Modeling  cognitive  systems  with  artificial  intelligence  &  fMRI.  Invited  presentation  at  BrainMap  ’98: 
Human  Brain  Mapping  and  Modeling,  December  7, 1998,  San  Antonio,  Texas. 

Other  Presentations 

Lerch,  J.,  Harter,  D.,  &  Gonzalez,  C.  (1998).  Time  pressure  in  real-time  dynamic  decision  making. 

In  the  Proceedings  of  the  Association  for  Information  Systems  Conference  (AIS,  Americas 
Conference,  1998),  Baltimore,  Maryland  on  August  14-16, 1998. 

Lerch,  J.,  Harter,  D.,  &  Gonzalez,  C.  (1998).  Individual  differences  in  real-time  dynamic  decision 
making.  In  the  Proceedings  of  the  Fourth  Conference  on  Naturalistic  Decision  Making,  W arrenton, 
Virginia.  May  29-31,  1998. 

Gonzalez,  C.  Instance-Based  Decision  Making  in  Dynamic  Environments:  Modeling  the  Learning 
Process.  Recognition-Primed  Decision  Making  Workshop.  Boulder,  CO.  October  23, 2001. 

Gonzalez,  C.  Instance-Based  Decision  Making  in  Dynamic  Environments:  Modeling  the  Learning 
Process.  Computational  and  Mathematical  Organization  Theory  Conference.  CASOS  2001. 

Pittsburgh,  PA.  p.  71.  July  6,  2001. 

Gonzalez  C.,  Lebiere  C.,  and  Lerch  F.J.  ACT-R  Learning  in  a  Real-Time  Dynamic  Decision- 
Making  Task.  Proceedings  of  the  Sixth  Annual  ACT-R  Workshop.  George  Mason  University. 

Fairfax,  Virginia.  August  6-9, 1999. 

fMRI  and  the  neuro-architecture  of  cognition.  Symposium  held  at  the  Psychonomics  Society,  November 
19-21, 1999,  Los  Angeles,  California. 

4CAPS:  Cortical  Capacity-Constrained  Concurrent  Activation-based  Production  System.  Presented  at  the 
ONR  Conference,  June  5, 2000,  Baltimore,  Maryland. 

The  brain-activation  dynamics  underlying  fluid  intelligence.  Presented  at  the  1st  Annual  Conference  of  the 
International  Society  for  Intelligence  Research  (ISIR),  November  30,  2000,  Cleveland,  Ohio. 

fMRI  and  the  neuro-architecture  of  cognition.  Presented  at  the  T.  J.  Watson  IBM  Research  Center, 
January  31, 2001,  Yorktown  Heights,  New  York. 


b.  Consultations:  The  project  investigators  consulted  with  Air  Force  personnel  at  Wright  Patterson  AFB 

(Glenn  Wilson)  and  Brooks  AFB  San  Antonio  (Shifflet,  Hall,  Regian). 

c.  Transitions:  N/A 
8.  New  Discoveries: 
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The  project  has  discovered  a  surprising  phenomenon  regarding  an  upperbound  on  cognitive  workload 
during  the  performance  of  two  concurrent  tasks:  The  total  amount  of  brain  activation  was  approximately 
similar  during  the  performance  of  one  versus  two  tasks,  indicating  that  a  given  task  is  allocated  much  less 
activation  if  it  is  being  performed  concurrently  with  another  task. 


9.  Honors/Awards: 

1 997-2002  NIMH  Senior  Scientist  Award 

2000  American  Association  for  the  Advancement  of  Science  Fellow 
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MJ  notes: 

Check  on  ONR  description  of  TOL  task 
Check  CV  for  other  possible  publications 
Proofread  for  sensibility 


